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Abstract 

We study neutrino mass generation in models with four chiral families of leptons and 
quarks and four right handed neutrinos. Generically, in these models there are three 
different contributions to the light neutrino masses: the usual see-saw contribution, 
the tree-level contribution due to mixing of light neutrinos with neutrino of the fourth 
generation, and the two loop contribution due to the Majorana mass term of the fourth 
neutrino. We study properties of these contributions and their experimental bounds. 
The regions of the parameters (mixings of the fourth neutrino, masses of RH neutrino 
components, etc.) have been identified where various contributions dominate. New 
possibilities of a realisation of the flavour symmetries in the four family context are 
explored. In particular, we consider applications of the smallest groups, e.g. SG(20,3), 
with irreducible representation 4. 
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1 Introduction 



There are various arguments in favour of the existence of a fourth Standard Model (SM) 
generation of fermions. 

- A fourth generation can alleviate the tension between the lower bound on the Higgs 
mass from LEP II and the fit of the electroweak precision data, which predicts a light Higgs 
particle [1]. Indeed, the mass splittings between the fourth generation fermions can lead to 
a negative contribution to the S parameter, which allows for a heavier Higgs. The flavour 
sector with four families has been thoroughly analysed in [2—5]. 

- The enlarged CKM matrix contains additional CP phases and naturally leads to more 
CP violation which can explain the deviations from the predicted SM values in some mea- 
surements in B physics [2; 6]. 

- A fourth generation has been suggested as an explanation of the anomalous like-sign 
di-muon charge asymmetry [7]. 

- A fourth generation makes viable electroweak baryogenesis, which is not possible in the 
SM. The introduction of a further generation leads to additional CP violating phases in the 
quark mixing matrix (CKM matrix), which are not constrained by experiment yet, e.g. see 
| l] for an analysis of the neutron electric dipole moment, and can lead to a large enough 
CP violation [9]. In addition, it has been shown, that a strong first order phase transition 
is possible within the SUSY version of a model with four SM generations (SM4) [] 0] as well 
as in a strongly coupled version with dynamical breaking of the electroweak symmetry [1 1] . 

- Being similar to top quark condensate models [12], dynamical electroweak symmetry 
breaking is possible in the context of four generations [13; 14]. 

- A fourth generation neutrino can contribute to the dark matter density of the Universe 
if an additional B — AL^ symmetry is introduced protecting the fourth generation neutrino 
from decaying and it couples to the three light generations via the new Z' to quarks [15]. 

- Under the assumption of minimal flavour violation, a fourth generation suppresses 
proton decay and enforces the R-parity in the context of the MSSM due to the mismatch 
of numbers of flavours and colours [16]. 

Significant interest to the fourth generation is also revived due to operation of the Large 
Hadron Collider (LHC). The LHC can provide a critical test of existence of the fourth 
generation: either discover or exclude it. (See [] 7] for a recent review and [18] for an 
earlier review.) Indeed, the LHC can test the region of fourth generation quark masses, 
300 — 800 GeV, [19], which covers the complete parameter space determined by the partial 
wave unitarity upper limit of 550 GeV for a quark doublet [ ] and the limit obtained 
in models of a strongly coupled fourth generation. The CMS Collaboration put a lower 
bound on the mass of fourth generation up- type quark t' of % > 450 GeV [21] and exclude 
fourth generation down-type quark b' in the mass region 255 GeV < my < 361 GeV at 95% 
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C.L. [22]. Existence of the fourth generation chiral leptons without fourth generation quarks 
looks rather unnatural and in fact this will require further complication of model to cancel 
the anomalies. 

The parameter space of the fourth generation can also be probed by looking for the 
Higgs signals [ ] (this has also been studied in the MSSM [24]). Currently, the Higgs 
boson with mass m# in SM4 with one Higgs doublet is excluded in the region 120 GeV < 
m H < 600 GeV at 95% C.L. by CMS [ ] and 140 GeV < m H < 185 GeV by ATLAS [26]. 
However, these bounds only apply in the minimal SM4 model with one Higgs doublet. 
They are weakened if (i) the Higgs production via gluon fusion is modified, e.g. by a colour 
octet [: ] or if the light Higgs in a two Higgs doublet model does not couple to the fourth 
generation [28], (ii) the search channels h — > WW*, ZZ* are modified, e.g. in a two Higgs 
doublet model [11], or (iii) the Higgs decays dominantly invisibly, e.g. into a light scalar, 
which can provide a dark matter candidate [ )] or into fourth generation neutrinos for light 
Higgs with ttlh < 170GeV [30]. Recently, the complete electroweak two-loop corrections 
to Higgs production via gluon fusion have been calculated and discussed in the framework 
of a fourth generation [31]. In the SM with four generations and one Higgs doublet, the 
Higgs bounds can be translated in a bound on the fourth generation fermion masses via the 
triviality and stability bounds [32]. 

Collider signals of the 4th generation have been studied which include signals of fourth 
generation quarks [19; 23; 33; 34], leptons [35-37], sleptons [38], signals of a strongly coupled 
generation [ 19] and a Z' [ ]. If the mixing of the fourth generation with the three SM 
generations is tiny, the particles of the fourth generation become long-lived [11]. The fourth 
generation quarks might even form long-lived bound states which can be produced at the 
LHC. The binding energies and sizes of those bound states have been calculated in [42]. 

Note that SM4 is constrained by the large Yukawa couplings running into Landau poles 
due to a quick renormalisation group (RG) evolution. The current experimental bounds 
require a cutoff scale A c < (10 2 — 10 3 ) TeV, unless there is a fixed point [43]. Similar results 
have been obtained in the SUSY context in [17; 44]. 

Properties of the fourth generation particles should differ from the properties of three 
known generations. The bound from the invisible Z decay width forbids further light gen- 
erations, especially additional neutrinos. The existing experiments give lower bounds on 
masses of fourth generation leptons (charged lepton and neutrinos) at the level of 100 GeV 
and an upper bound on the mixing parameters 0.04 — 0.08. 

The generation of neutrino mass in models with four fermion generations has been 
explored in several publications. The simplest model with four SM generations and usual 
massless neutrinos at tree level has one right-handed (RH) neutrino [45]. An explanation 
of neutrino masses in terms of the usual see-saw mechanism [ 16] , however, requires at least 
three RH neutrinos. Since the fourth generation neutrino should be much heavier than the 
three SM neutrinos, several authors suggested its pseudo-Dirac nature [45; 47; 48]. The light 
neutrino masses can be generated by two loop diagrams with two W bosons exchange in the 
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framework of five SM generations [49]. There are studies of the leptonic flavour structure 
in SM4 with discrete flavour symmetries Z 4 [50] and A 5 [51]. Moreover, the leptonic flavour 
structure have been explored in extra-dimensional 4 generation models [52]. 

The origin of neutrino masses or the flavour structure, and especially the number of 
chiral SM generations are not understood within the Standard Model. 

In this paper, we present a comprehensive study of the neutrino mass generation in 
the SM model with four fermionic generations including one RH neutrino per generation. 
We will restrict ourselves to a non-SUSY model. However, our results can be directly 
generalised to the SUSY case. Besides the usual see-saw contribution, we calculate and 
study the contributions from tree level mechanism related to mixing of the fourth neutrino 
with the light ones, and from the two W— boson exchange at two loop. We explore possible 
flavour symmetries in the context of SM4. We study the smallest discrete group with a four- 
dimensional representation and explore flavor structures that appear in the most economical 
scenarios. 

The paper is organised as follows. The contributions to the neutrino mass matrix from 
three different mechanisms are computed in Sec. 2. In Sec. 3, we consider existing bounds 
on fourth generation leptons, in particular, from the neutrinoless double beta decay. We 
find the regions in parameter space in which different mechanisms dominate. We explore 
possible realisations of flavour symmetries in context of SM4 in Sec. 4, and conclude in 
Sec. 5. In the Appendix we present the group theoretical details of the smallest group with 
a representation 4: SG(20, 3) = Z 5 Z 4 . 



2 Contributions to Neutrino Masses 



We consider the extension of the SM by one RH singlet fermion per generation, N k , k = 
1, 2, 3 for the first three generations and iV 4 for the fourth generation. We consider two SM 
Higgs doublets, one coupling to neutrinos Hi and one to charged leptons H 2 . The case with 
a single Higgs doublet is obtained by identifying H = H\ and H c = H 2 . In the flavour 
basis, the leptons have the following couplings 



-£ = Y ak e a H 1 N k + Y c J 



t H 1 Nt+Y E rf E H 2 N k + Y E 4,hH 2 Nt + ^M k ]tfN k + ^MtNZN4 + h.c., 

(1) 

where £ a , a = e, /x, r and £ E denote the light and the fourth generation left-handed lepton 
doublets, respectively. These couplings lead to the following neutral fermion mass matrix 
in the flavour basis (y a v E iV 4 iV fc ) 

/ f L 
... rriEA 
M 4 



M 



V 



m \ 

fl 


M } 



(2) 
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We take the RH neutrino mass matrix to be diagonal: M = diag(Mi, M2, M3). The 
complete Dirac mass matrix consists of the following components: 

- the Dirac mass matrix of the light SM neutrinos, m a k = ^a^ew ( a = e -> A* 5 T an d 
k = 1,2,3) with v^ w = (Hi) being the vacuum expectation value (VEV) of the Higgs Hi 
coupling to neutrinos, 

- the Dirac mass of the fourth generation: m# 4 = Y^f^, and 

- the column Jl which gives mixing of the fourth generation with the light ones fia = 

The neutrino Dirac mass can be written as 

C/ L diag(mi) U R . (3) 



According to (2) the left-handed (LH) mixing matrix elements of the fourth generation 
with the three light generations are given approximately by 

(U L ) a4 * (4) 

which are bounded by experiments to be smaller than 0.04 — 0.08 [53]. Similarly, the RH 
mixing matrix elements can be estimated as 

(Un) M (5) 
rriE4 

provided that m^ 4 dominates the Dirac mass matrix. 

Decoupling of the RH neutrinos, A^ 12i 3, in Eq. (2) leads to the effective mass matrix in 
the basis (v a , u E , iV 4 ) T : 

-mM- 1 m T -mM- l f R f L 
M' = I ... -flM- l f R m E4 I . (6) 

M 4 

At this level the three active neutrinos acquire the usual see-saw contributions associated 
to the three heavy RH neutrinos. Notice that in the limit Jr = further decoupling of v E 
and iV 4 in Eq. (6) gives zero contribution to the light neutrino masses in spite of the fact 
that u a interacts with iV 4 . 



2.1 Tree-level Mechanism from Mixing with Fourth Generation 

Depending on value of M 4 there are two extreme cases for the masses of the fourth neutrino: 
the see-saw case, M 4 >me 4 and the pseudo-Dirac case M 4 Cm^. 
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1). In the see-saw case after decoupling the fourth RH neutrino we obtain the 4x4 
neutral fermion mass matrix in the basis (u a , v E \. 



Ma Ma 

™2 



(7) 



M 4 



where we have neglected the see-saw contributions of the first three RH neutrinos to the 
fourth row and column since M4 <C The mixing matrix elements can be estimated as 

UaA ~ J^L ~ (jj L ) a4 . (8) 

Further decoupling of z/4 leads to cancellation of the leading order contribution of the fourth 
generation to the light 3x3 neutrino mass matrix. Non-zero contributions are generated 
by the next-to-leading order see-saw effect [ ]. Therefore, in this limit, a fourth generation 
does not give a substantial tree-level contribution to the light neutrino mass. 

2) In the pseudo-Dirac case, M 4 <mg4, under the assumption of fjn- <C m Ei a block 
diagonalisation of matrix (6) (i.e. decoupling the pseudo-Dirac pair (1/4, N 4 )) leads to 

m v ~ m ss + m tree (9) 

with m ss = —mM~ 1 m T and 



m tree 



{mM- l f R f T L ) + (...) T . (10) 



The contribution m tree is linear in the light Dirac mass m and therefore can be considered 
as a new realisation of the linear see-saw [55]. Up to high order corrections the total mass 
matrix in Eq. (9) can be rewritten as 

Thus the total mass matrix can be considered either as a combination of linear and ordinary 
type-I see-saw (as in Eq. (9)) or as a type-I see-saw with a modified Dirac neutrino mass 
term. 



We can rewrite the contribution of the fourth generation to the a/3 matrix element of 

m u as 

<T - m EA J2 ^(UMUrU + (a p) . (12) 

k k 

Hence, m^ r | e is suppressed by the left- and right-handed mixing in addition to the usual 
see-saw type factor. As the leading contribution has rank one and the sub-leading ones 
are suppressed, it can only generate one mass scale and the ordinary see-saw contribution 
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cannot be completely neglected. We can compare the contributions of the fourth generation 
via the k th RH neutrino with the see-saw contribution of the k th RH neutrino to the light 
neutrino mass matrix as 

(m| s ) a/3 m kf j/m E4 

Therefore the tree level contributions via the k th RH neutrino dominates over the seesaw if 

(U R ) M (U L ) a4 > m k p/m EA . (14) 



2.2 Two-Loop Mechanism 

If the components of the fourth generation neutrino are Majorana particles, they induce 
a Majorana mass term for the light neutrinos at two loop level, m loop , see Fig. 1(a) [56]. 
We will consider the system of five neutrinos (u a , ve, N±) after decoupling of the heavy RH 
neutrino components. The tree level mass matrix is then given by Eq. (6) and we neglect 
the see-saw contributions to the v a — v E as well as v E — ve elements. 

The expression for the two-loop generated Majorana masses given in Eqs. (21,22) of [56] 
can be rewritten in the flavour basis as 

loo g 4 

m A°B i-rn E4 M4m\m B (U L ) A4 (U L )B4lAB • (15) 

Here the indices A, B — e, fi, r, E run over four generations, and mg are the charged 
lepton masses, g is the SU(2) gauge coupling and mw is the mass of the H^-boson. The 
integral Iab equals 

d 4 p f d 4 q p ■ q 1 1 



W I (2vr) 4 7 (27r) 4 (p 2 -^)(g 2 -m|)(p + g) 2 -m^ 1 (p + g) 2 -- 2 



m 



N2 
1 



p 2 q 2 4 p 2 — rriyy q m\y 



(16) 



Here rn.jvi,2 are the eigenvalues of the mass matrix of the fourth generation neutrino states. 
According to Eq. (15), the two loop generated masses depend on the mixing of the fourth 
generation neutrino with the light neutrinos, Uaa, the Dirac mass of the fourth generation 
neutrino, uieai and the Majorana mass of the RH neutrino, M 4 . 

There are two different two-loop contributions to the mass matrix of light neutrinos: 
(i) the direct one which follows from Eq. (15) for A,B = e,fi,r, and (ii) the contribution 
via the EE-element, the Majorana mass of Ue, rriEE generated in 2 loops. As we will see, 
the latter dominates due to hierarchy of the charged lepton masses: mg» rn e ^ T . In fact, 
this contribution can be computed in the approximation of vanishing charged lepton masses 
m^ T = 0. 
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Let us compute the second contribution in the pseudo-Dirac case when the masses of 
neutrinos of the fourth generation equal m^i — itln2 — m i and the splitting between them 
given by M 4 is small: M 4 <C 777.4. According to Eq. (15) the m EE element is given by 



in 



lo °P ~ _ „4_-4_2 S/f _4 ( TT \2 
EE ~ 



g m w m i M 4 m E (U L ) E J EE . (17) 



In the limit mw <C friA <C m E the integral I EE equals approximately * 



I /tt 2 „ , m\ 



Now we have the mass matrix (6) with non-zero elements in the fifth row and column and 
non-zero i?.E-element. Decoupling of the fourth (pseudo-Dirac) neutrino {i.e. the see-saw 
diagonalisation with v e ,Na heavy block) contributes to the masses of the light neutrinos: 

?T - 



,i°°p ~ ^loop [l/l 

2 lo 

l EA ~ m EE 

which in the case m Ei 3> rn EE ? M4 leads to 



m loop ~ m|°? , (19) 

mL - mZ°p Ma 



m !oo P „ m ^i^^- = m}°°£(U L U(U L U • (20) 



m E4 m E4 

The resulting contribution to the light neutrino mass matrix is 



m ioo P „ _^ M ^j EE{UL y Ei(UL)ai(UL)pA 



w 



9' (%>„ f«_l_ 2 + ln <\ r_^i Mi (Ul) (Ul) . (21) 
V 3 m E J p 



(4tt) 



The mass matrix formed by the loop contribution via m} E ° E (21) is singular (rank 1) with 
the unique non-zero eigenvalue 



>o P _ 9 4 m%_ Mim 2 



m w 



tEE^2(U L ) 2 aA 

a 



(22) 



When the masses of three SM charged lepton are taken into account, the two loop contri- 
bution obtains full rank. However due to a strong hierarchy of these masses the two loop 
contribution cannot explain neutrino masses by themselves [49]. The contribution (21) dom- 
inates over the direct two loop contribution (15) due to three known SM leptons. Indeed, 
the ratio of the two equals: 

^hL, (23 ) 
m% I ee 
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w 




(a) Two W exchange contribution to light (b) Rank changing two loop diagram con- 

neutrino mass matrix. tributing to Majorana mass term. 

Figure 1: The two loop diagrams which describe contributions to Majorana neutrino masses. 
Shown are the diagrams for the light neutrinos (left), and for the heavy RH neutrino (right). 

The flavour structure of the loop contribution is determined by the mixing matrix elements 
(Ul)oA. Note that the results considered here can be immediately obtained by computing 
diagrams with the would-be Goldstone bosons, then all the substantial quantities arise from 
vertices. 

In the see-saw case for the fourth generation masses, M4 3> the two-loop contribu- 
tion is suppressed by fia/m-Ei only, because the direct Majorana mass term M 4 is large. 
Consequently, the mixing between the fourth generation and the three light SM generation 
has to be small. 



2.3 Radiative Generation of Fourth Generation Singlet Majorana 
Mass 

Similarly to itlee considered in the previous subsection, the RH Majorana neutrino mass 
M4 can be generated at the two loop level. The relevant (rank changing) two loop diagram 
is shown in Fig. 1(b), which results in the following expression for the mass in the MS 
renormalisation scheme 

M'° op = 2 ^ Y(Y^Y) ik (yty) Mk (- + -+\n . (24) 

Here i,j = 1, . . . , 4 and the Higgs mass has been neglected. The RG produced fourth gen- 
eration Majorana mass can be estimated (neglecting diagrams with light charged fermions) 

as 

M l ;° P * j^y £ [(U r )UUr)eA 2 Mi In ^ (25) 

with A being the high scale, at which the theory is defined, and 7/4 = m E4i /v^ w being the 
neutrino Yukawa coupling of the fourth generation, which dominates over the other Yukawa 

*See the appendix of [49] for the evaluation of this integral. 
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couplings. An estimate can also been be found in [57]. In addition to the RG running, the 
finite part of the counter term leads to a scheme dependent threshold correction The main 
contribution comes from the diagrams with the charged lepton E. According to Eq. (25) 
M4° op oc TnjjUft and therefore it quickly decreases with m 4 . For Ur ~ Ul inspired by the 
L-R symmetry and a single Higgs, i.e. Hi — H and H2 = H c , the mass can be estimated 
as M 4 = 1.0 GeV for U R = 0.001, m 4 = 400 GeV, M t = 10 8 GeV and A = 10 M<. 

Neglecting an accidental cancellation, we expect that the fourth generation Majorana 
mass, M4, has at least the size of the radiatively generated contribution given in Eq. (25): 
M 4 > M] oop . Using M l ° op only, we can estimate the size of the contributions to the neutrino 
masses using Eq. (21): 



m 



loop yig 4 {U L f m fir 2 m\ \ m\m\ ^ a M, 

j - - 4(87r2)4 (y ~ 2 + ln < ) -<r (c/ ^ (f/ ^ g [{Ur) - {Ur) ^ ] m ln t 

(26) 

Effectively it is generated at four loops level. According to Eq. (26), and since (Ul)ea ~ 
(Ur) ea ~ 1 the bound on the neutrino mass scale leads to an upper bound on the combina- 
tion \(U R )i4 : (UL) a i\ 2 Mi. The bound strongly depends on m 4 = y 4 fE W and m^. 

Fig. 2 shows the iso-contours of the two loop contribution to the light neutrino masses in 
the plane of mixing angle \U a ^\ and the fourth generation Majorana mass M 4 . The equation 
for these contours is given by Eq. (21) which can be rewritten as 



\(Ul)J 

where 



loop 



CoMa 



m 2 w (27) 



m^m 4 ' 



(4tt) 4 V 3 m' E 



c a = - » ;;r/ 4 v-2+in^ . (2 



The vertical lines in Fig. 2 indicate the Majorana mass M 4 generated by the two loop cor- 
rection (25) for y4 = m 4 /t>E W . So, 1(^)^1 oc 1/VM 4 , and furthermore the mixing becomes 
small with increase of mg. Values of mixing parameters (Ul)^ at the level achievable by 
the direct searches can be obtained only for very small RH neutrino masses: M 4 < 1 keV. 
On the other hand M 4 of the size ~ 1 GeV requires (Ul) 04, ~ 10 -5 . 

The bound on the light neutrino mass scale strongly constrains the combination |(C/l)q 4 M 4 | 
Barring accidental cancellations, we expect the fourth generation RH neutrino mass to be 
at least of the scale generated by the two loop diagrams. Under this assumption, the mixing 
parameter |?7 q4 | should be smaller than the value at the intersection of the vertical line with 
the lower border of the excluded shaded area. 



^Notice that this result for the radiatively induced RH neutrino mass is rather general and valid beyond 
the four generation context. It is particularly interesting in case of the nearly singular see-saw when one of 
the mass eigenvalues is considerably smaller than the remaining ones. 
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lcr 6 icr 3 io _0 

M 4 /GeV 

Figure 2: Iso-contours of the two loop contribution to light neutrino masses in the (L r x,) Q , 4 — 
M4 plane for different values of the Dirac neutrino mass, TO4, and charged lepton mass, 
rriE- The black solid lines correspond to 777,4 = 400 GeV and itle = 600 GeV, while the blue 
dashed lines correspond to m 4 = 900 GeV and = 1 TeV. The shaded region is excluded 
by the cosmological bounds on the neutrino mass. The green shaded areas are excluded by 
the bounds on the mixing angles U a 4 (see Sec. 3.1). The vertical lines correspond to the 
Majorana mass M4 induced at the two loops level for a RH neutrino mixing \Ur\ = 10~ 4 
and RH neutrino mass M = 100 TeV with the cutoff scale A = 1000 TeV. 



These two loop corrections do not exist in SUSY due to the non-renormalisation theorem 
for the superpotential [ 8] . However, if there are RH neutrinos with a mass below the SUSY 
breaking scale, there are quantum corrections. The larger particle content in the SUSY 
version compensates for the smaller logarithms coming from the RG corrections (see e.g. 
[59] for two loop corrections to the light neutrino mass matrix). 

3 Phenomenology of Fourth Generation Neutrinos 

3.1 Existing Bounds on Fourth Generation Leptons 

Let us summarise the bounds on masses and mixing of the fourth generation particles which 
we will use in our analysis. The collider searches give [60] 

m E > 100.8 GeV , m N > (80.5 - 101.5) GeV . (29) 
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The range of values for the lower bounds on in Eq. (29) originates from different search 
channels, N — > W* + (e, /i, r) under the assumption of a 100% branching ratio in a given 
channel. The bounds depend also on the nature of the neutrino: for Majorana neutrino 
they are about 10 GeV weaker than for Dirac neutrinos. The bounds rely on the assumption 
that only one heavy neutral lepton can be produced. A recent reanalysis [ ] shows that the 
bounds can be relaxed when two heavy neutral leptons are accessible, like in the framework of 
pseudo-Dirac neutrinos. Under the assumption of a mass splitting M N2 —M Nl > 10 GeV be- 
tween the two heavy neutral states N 1>2 , the study of e + e~ ->• Z* -> NiNj ->■ IW*IW*Z*°> 1 > 2 
leads to the bounds 62.1 GeV(W*r), 79.9 GeV (W» and 81.8GeV(W*e). The number of 
Z*-bosons depends on the number of produced N 2 via N 2 — > N%Z*. In our study, we mainly 
consider smaller mass splittings M^ 2 — , where the branching ratio of N 2 — > IW* dom- 
inates over the one of N 2 — > N\Z* . This leads to an interference between N\^ 2 — > IW* and 
we expect the bounds to become weaker. 

A study of the sensitivity of the Tevatron to a fourth generation neutrino [37] shows that 
it can put a lower bound m N > 175 GeV and has a 3a discovery potential for m N < 150 GeV 
with 5 fb -1 . The LHC can exclude fourth generation charged leptons up to 250 GeV [35]. 

The electroweak precision tests (which include quark mixing but neglect leptonic mixing) 
constrain the mass splitting between the fourth generation leptons [ ] 



indicating that the masses of the fourth generation leptons should be of the same order of 
magnitude. 

The leptonic mixing angles are constrained by searches for the radiative /i~ and t~ 
decays, £{ — > ^7, as well as by kaon and pion decays. The limits given in [53] read 



There is an even stronger bound on U* 4 U e 4 from the /x — e conversion: \U* 4 U e 4\ < 0.4 ■ 10 4 
for m N > 100 GeV [61]. 

The influence of mixing of light generations on the masses of the fourth neutrino can be 
neglected. We can estimate the maximal allowed value of M4 which is realized in the see-saw 
limit, M 4 3> m 4 , as M 4 = vn^Jm^. Using the unitarity upper limit on m 4 < 1.2 TeV [ 
and the LEP exclusion limit for an additional neutral lepton ~ 100 GeV we find 



tue — m N \ < 140 GeV, 



(30) 



t^PMNS 



/ . . . < 0.073 \ 

< 0.045 

< 0.072 

\ < 0.092 < 0.092 < 0.092 > 0.9958 / 



(31) 



M 4 < 14 TeV. 



(32) 
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3.2 Neutrinoless double beta decay and Cosmological Bounds 

All three main mechanisms of light neutrino mass generation are essentially of the see-saw 
type and the (3 (3 0u -decay proceeds via the neutrino exchange only. Therefore, we can apply 
here the results of [6 ]. Following [6 ], we separate the contributions to the amplitude of 
the decay from a heavy mass eigenstates with a mass rrij 3> m n ~ 100 MeV, and from the 
light neutrino mass eigenstates with mass rrii <C m n : 

light heavy 

AocJ2 mUliM^irrii) + ]T mi U 2 eI M^( mi ) , (33) 

i I 

where the masses my and mixing angles U ei , U e j are defined by 

?7*diag(m 1 , . . . , m 8 )U j = M (34) 

with M. being the 8x8 neutral fermion mass matrix. 

The two loop direct contribution to the m ee element in the flavour basis is negligible 
being proportional to m\. Therefore, according to Eq. (2) m ee f=s 0. In terms mixing angles 
and mass eigenstates defined in Eq. (34) this condition can be expressed as 

light heavy 

£ tt^S 0. (35) 

i I 

The nuclear matrix elements M 0ul3lB in Eq. (33) include neutrino propagators: 

{ 2 1 2 ~ \, for m ; C m w 
— L-?~-^> for mj>nv ' 

where m n ~ 1/rjv is the pion mass, which gives the inverse size of the nucleus radius, rjy. 
Therefore M "^ practically does not depend on the mass of the exchanged light neutrinos: 
M 0l//3/3 (mj) m M 0ul3l3 (0). For heavy neutrinos the matrix element decreases as M 0v ^^{mi) oc 
mj 2 . Consequently, the ratio of the matrix elements 

Using relation (35) we can rewrite the amplitude Eq. (33) in the following way 

light heavy 

A oc J2m t U 2 ei M°^(0) + mjU^M^imj) 

i I 
heavy 



mjU^M^imj) - M°^(0)) 

heavy light 

- mjU^M^iO) = Y^niiUlM ^^) , (38) 
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where we used M 0v/3/3 (m 7 ) < M°^(0) « M 0i/W (m i ) in the first and third line and Eq. (35) 
in lines two and three. Hence, the dominant contribution to f3f3o u -dec&y is from light neu- 
trinos. 

Thus, the bound from (3(3 0u -decay is reduced to the bound on the effective Majorana 
mass of the electron neutrino due to light neutrinos only. That is, the /3 /3 0u -decay gives a 
bound on the light neutrino masses, which has also been pointed out in [49], and as far as 
this bound is satisfied, no other bounds on the model appear. Hence, /3/3oi,-decay restricts 
the model via the light masses only. 

Notice that in the discussion of /S/^-decay in [63], the light contribution has been 
neglected. 

At the moment, cosmology gives even a stronger bound on light neutrino masses than 
the /3/3oi/-decay. We took m < 0.15 eV for an individual neutrino as reference value which 
originates from the bound Yl m i ~ eV [64]. 

3.3 Comparison of Different Contributions to Neutrino Mass 

As we have found in the previous section, in models with four families of fermions, generically 
there are three contributions to the light neutrino masses from three different mechanisms: 
(i) the usual see-saw type-I, m ss ; (ii) the tree-level contribution m tme due to mixing of the 
light neutrinos with z/4 is essentially another see-saw, it is linear in the usual Dirac mass 
matrix; (iii) the 2-loop contribution induced by the Majorana mass term of the neutrino 
of the fourth family, m loop . These three contributions have different flavour structures but 
partially correlate. For a given Dirac mass matrix of light neutrinos: m ss = m ss {M k ) ) 
m tree = m t™(M k ,U R ,U L ) (Eq. (12)), and m loo P = m loo P(M k ,U R ,U L ) (Eq. (21)). 

In what follows, we will consider these contributions in the case of a vanishing fourth 
generation Majorana mass term M 4 at tree-level. The mass M 4 is constrained on the one 
hand by the invisible Z-decay width and on the other hand by the bound on the neutrino 
mass which is induced at two loop (see Fig. 2). Furthermore, we restrict ourselves to a 
single Higgs doublet, i.e. Hi = H and H 2 = H c . Similar conclusions apply in a two Higgs 
doublet model. The main difference is an increased neutrino Yukawa coupling, for fixed 
values of masses. This leads to a larger loop contribution to neutrino masses. The following 
discussion does not depend on the Higgs mass m#, as long as it is negligible compared to the 
heavy RH neutrino masses Mi, i = 1,2,3. If they are of a similar magnitude, the expression 
(25) for M4 will change, but it remains valid as an order of magnitude estimate. Hence, the 
following conclusions are also valid for a heavy SM Higgs m# > 600 GeV. 

We will first discuss the "1+1" generation case: one light generation and the fourth 
generation In this case we have one light neutrino with Dirac mass m = m a i and one 
very heavy RH neutrino with mass M = Mj. We can introduce a single parameter which 

^The results can be directly applied to one specific matrix element in case of 3+1 generations. 
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characterises mixing of the light neutrino with the neutrino of the fourth generation: 



£ = (U L ) a4 (U R ) i4 . (39) 
In terms of this parameter the tree level contribution (12) can be written as 



The loop contribution (26) is then 



m 



loop 



2m 

^ = mEA M ^ (40) 



C loop m|m^ 2 Mln^ , (41) 



A 
where 

r loop = 9 ' 1 (— - 2 + In ™*\ (42) 

~ 4(8^)^^13 Z + [n ml) 

and we have taken into account that (Ul)e4 ~ {Ur)e4 ~ 1- Let us underline that mix- 
ing parameters of the fourth neutrino enter the contributions only in the combination £. 
The other relevant parameters are the, m^, M and m. Relative contributions of different 
mechanisms depend on values of these parameters. 

In Fig. 3 we show the iso-contours of different contributions to neutrino mass in the 
£ — M plane for fixed values of m#, m 4 and m. The equations for these contours can be 
readily obtained from Eq. (40) and Eq. (41). For a given value of m tree we find from Eq. (40) 
the following dependence of £ on M: 

£trce = M mtme _ ( 43 ) 

2mm ea 

That is, £ tree linearly increases with M; it is proportional to m tree and inversely proportional 
to m. The iso-contours of the tree-level contribution correspond to the black dashed lines. 

From Eq. (41) we obtain the analytic expression for iso-contours of the loop contribution 
(blue solid lines in Fig. 3): 



M 2 

^•loop 



^loop 



JMlnf 



' (44) 



£71oop m | m 6 

For M oc A this equation gives £ 1o °p oc 1/ \/M. 

The usual see-saw contribution, m ss = —m 2 /M, does not depend on £ and the corre- 
sponding iso-contours are just vertical red dotted lines in the plot of Fig. 3. According to 
Fig. 3, the loop contribution dominated for large values of M and small values of £. The tree 
level contribution is larger for small M and large £, whereas the usual see-saw dominates 
in the range of small M. For m = 31 MeV (see Fig. 3(a)) the allowed region is M > 10 7 
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10° 10 3 10 6 10 9 10 12 10 15 10° 10 3 10 6 10' 10 12 

M/GeV M/GeV 



(a) m = 31 MeV (b) m = m e 

Figure 3: Iso-contours of different contributions to light neutrino masses (numbers at the 
curves) from different mechanisms in the £ — M plane for m 4 = 400 Ge V and = 600 Ge V 
and two different values of the Dirac mass m. Black dashed lines correspond to the tree-level 
contribution, blue solid lines to the loop contribution and red dotted ones to the usual see- 
saw contribution. Note that the Higgs mass m# has been neglected in the calculation of the 
radiatively induced M 4 . Hence for M ~ ra#, the loop contribution can only be considered 
as an order of magnitude estimate. 



GeV and £ < 10~ 6 . The tree-level contribution is negligible in this region and the total neu- 
trino mass is determined by an interplay of the usual see-saw and the loop contributions. 
Furthermore, the see-saw dominates at smaller M and £. 

With decrease of m, the relative contributions of different mechanisms change: the iso- 
contours of m loop do not move, the see-saw lines shift to smaller M as M oc m 2 , whereas the 
iso-contours of tree level contribution shift as M oc m, i.e. weaker. Therefore, the tree-level 
contribution becomes important and can dominate for a small Dirac mass m in the range 
of a small RH Majorana mass M and relatively large £. With increase of 777.4, £ loop oc l/m\ 
decreases faster than £ tree oc 1/777,4. Therefore the tree level contribution becomes substantial 
and the allowed region shifts to smaller £. Also with increase of the charged lepton mass 
m e the loop contribution increases. 

In Fig. 4 we show the iso-contours of different contributions to light neutrino masses 
in the m 4 — M plane for fixed m = 31 MeV and iue = 777.4 + 200 GeV. As in the Fig. 3, 
the contours of the tree level contribution can be obtained from Eq. (40) and the contours 
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10° 10 3 10 6 10 9 10 12 10 15 10° 10 3 10 6 io t; 10 12 



M/GeV M/GeV 

(a) £ = 1CT 7 (b) £ = 10~ 9 

Figure 4: Iso-contours of the contributions to light neutrino masses (numbers at the curves) 
from different mechanisms in the 777,4 — M plane for two different values of the mixing 
parameter £. The Dirac mass m is fixed to m = 31 MeV and the fourth generation charged 
lepton mass m E is fixed to be 200 GeV larger than m 4 : mg = m 4 + 200 GeV. Black dashed 
lines correspond to the tree-level contribution, blue solid lines to the loop contribution and 
red dotted ones to the usual see-saw contribution. Note that the Higgs mass m# has been 
neglected in the calculation of the radiatively induced M 4 . Hence for M ~ m#, the loop 
contribution can only be considered as an order of magnitude estimate. 



of the loop contribution can be read off from Eq. (41). The usual see-saw contribution, 
m ss = —m 2 /M, does not depend on m 4 and the corresponding iso-contours are just vertical 
lines. The iso-contours are coloured in the same way as in Fig. 3. 

Since the loop effect alone cannot explain neutrino data, the other contributions (the 
see-saw or /and tree level) should be present and without strong suppression. 

For large RH neutrino masses the loop contribution to the light neutrino masses domi- 
nates over the usual see-saw contribution as well as the tree-level contribution. In particular, 
the two loop contribution is incompatible with RH neutrino masses close to the GUT scale, 
unless the fourth generation effectively decouples, i.e. £ < 10~ 12 . Hence, the combination 
of LH and RH mixing angles of the fourth generation with the three light SM generations is 
highly constrained by this contribution. The tree-level contribution of the fourth generation 
is only relevant for small RH neutrino masses (particularly below 100 TeV for the values 
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fixed in Fig. 3) and therefore small Dirac masses. 

Note that contributions of the fourth generation to the light neutrino masses (both tree 
level and loops if M 4 = Mj oop , see Eq. (12) and Eq. (26)) are proportional to mixing of 
the RH neutrinos. Therefore in the limit Ur — > 0, the light neutrino masses are generated 
by the usual see-saw mechanism and the left mixing of the fourth generation with the first 
three generations can be large: at the level of the upper bounds. 

3.4 3 + 1 Generation Case: an Example 

The results "1+1" generations presented in the previous section can be also used in the 
analysis of the (3 + 1) case. Here we present an example, where the two loop contribution 
of the fourth generation is essential for neutrino masses. 

We assume that three massive RH neutrinos have the common Majorana mass Mo = 
10 9 GeV and the Majorana mass of the fourth RH neutrino is zero at tree-level. In the 
flavour basis, the neutrino Dirac mass matrix of the three light SM generations is given by 
the democratic mass matrix with the common mass scale of m = 31 MeV, so that the usual 
see-saw contribution equals 



We take the Dirac mass of the fourth generation to be m 4 = 400 GeV and the fourth 
generation charged lepton mass, m# = 600 GeV. For the RH mixing of the fourth generation 
(U R ) a4 ~ 1.08-10" 4 and the left-handed mixing (U L ) ~ 1.08-10" 4 -(0.15, 1, -1), the two-loop 
contribution with a cutoff scale A = 10 Mq equals 



The tree-level contribution of the fourth generation is of the order of 10 6 eV, and therefore, 
negligible. 

The loop and see-saw contributions have rank 1 and their combination leads to a strong 
normal mass hierarchy with the mass splittings Am^ = 2.50 ■ 10 -3 eV 2 and Am^ = 
7.41 ■ 10" 5 eV 2 , and mixing angles sin 2 ^ 2 = 0.330, sin 2 #i 3 = 0.013, and sin 2 # 23 = 0.510 in 
agreement with observations. 




(45) 



m 




(46) 
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4 The Fourth Generation and Symmetries 



In spite of many efforts to explain the observed features of lepton mixing using various 
discrete flavour symmetries, no convincing model has been proposed so far (see [65] for recent 
review). In this connection, we will explore whether the existence of 4th generation can help 
in the realisation of discrete flavour symmetries. Existence of four generations of fermions 
can be explained if the flavour symmetry group has the lowest irreducible representation 4 
(apart from singlet representations). The key feature here is very small mass of one right 
handed neutrino, M 4 <C which can be a consequence of a certain symmetry. In this 
connection, a natural question is whether the same symmetry which leads to M 4 -C M& can 
produce certain flavour structures for the three light generations? 

In the following, we will present some general results on model-building in the four 
generation context and then focus on the simplest symmetry group in detail. 

4.1 General Comments 

As we have shown in Sec. 3, the following features are important for model building: 

• The RH neutrino mass matrix should be nearly singular. Three massive and one 
(almost) massless RH neutrino are required. 

• The two loop contribution to the light neutrino mass matrix has rank 1 and its flavour 
structure is given by Jl, that is, by the LH mixing of the 4th generation with the 
three light SM generations. 

• The tree-level contribution to m v produced by mixing of light neutrinos with fourth 
generation neutrino is negligible compared to the two loop contribution for large RH 
neutrino masses, and it becomes important only for small as it can be seen in 
Fig. 3 and Fig. 4. 

The simplest possibility to obtain a pseudo-Dirac structure for the fourth neutrino is to 
impose the conservation of the fourth generation lepton number, L4. This implies decoupling 
of the 4th generation. Breaking of the lepton number symmetry is then needed to mix the 
4th neutrino with the light neutrinos. This leads to the tree level contribution m tree , and 
possibly to the generation of a 4th generation Majorana mass term, which in turn produces 
a two loop contribution. The spontaneous breaking of the global U(1)l 4 symmetry results in 
a Goldstone boson (Majoron). This Majoron is not dangerous, because it couples directly to 
the fourth generation only and its coupling with the three light SM generations is suppressed: 

g a p ~ U ai Up 4 M 4 /m^. 
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Notice that this coupling has a similar dependence on V a /JJ p/JsA^ as the 2-loop contribution. 
For M 4 = 1 GeV and (U L ) ^ 4 ~ 10 7 , one obtains g a p < 10 8 which satisfies limits on 
the Majoron couplings [66]. Different values of M 4 lead to a similar limit on the coupling 
g a p due to the similar dependence on [/ a4 [/g 4 M 4 . The U(l)i 4 symmetry cannot be gauged, 
unless additional particles are introduced to cancel the anomalies. 

An abelian symmetry can only forbid certain terms in the mass matrix and produce a 
mass hierarchy but cannot lead to relations between different elements of the matrix. In 
this connection, we consider non-abelian groups with the lowest non-trivial irreducible rep- 
resentation 4. This (i) explains existence of four generations, and (ii) opens up a possibility 
to obtain certain flavour structures. 

We use the SmallGroups catalogue of GAP [67] to obtain the groups with irreducible 
representation 4 in a systematic way. They are denoted as SG(iV, m), where N is the order 
and m is the index of the group in the SmallGroups catalogue. We find that the smallest 
group with a 4 dimensional representation is SG(20, 3) = Z5 x^Z^ It does not contain other 
non-singlet representations besides 4 and has order 20. Hence, it is much smaller than A5, 
which has been studied in [51]. The next groups with a real four-dimensional representation 
are of order 32: 

SG(32, 6) S ((Z 4 x Z 2 ) x v Z 2 ) x^ Z 2 SG(32, 44) = (Z 2 x Q 8 ) x v Z 2 

SG(32, 7) = (Z 8 x^ Z 2 ) x^ Z 2 SG(32, 49) (Z 2 x D 4 ) x^ Z 2 

SG(32, 8) = (Z 2 x Z 2 ) x v (Z 4 x Z 2 ) SG(32, 50) = (Z 2 x Q 8 ) x v Z 2 

SG(32,43) S (Z 2 x D 4 ) x^ Z 2 , (47) 

where the defining homomorphism ip/ip of each semi-direct product is not specified explicitly. 
The smallest groups with a complex four-dimensional representation are of order 60: 

SG(60,6) = Z 3 x (Z 5 x v Z 4 ) SG(60,7) = Z 15 x v Z 4 SG(60, 8) = S 3 x D 5 . (48) 

In the following, we will concentrate on the smallest group SG(20,3). 

4.2 The Smallest Group: SG(20, 3) ^ Z 5 xi^ Z 4 

The smallest group with a four- dimensional representation, SG(20, 3), is the Frobenius group 
of order 20, with presentation 

(s,t\s 4 = t 5 = l,ts = st 2 ) , (49) 
which can be considered as a subgroup of S$ generated by 

((2, 3, 5, 4), (1,2, 3, 4, 5)) . (50) 
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The decomposition of the Kronecker product lj x lj equals 
( II 12 l3 14 ) X 



12 

V i4 / 



/II 12 13 14 

... l! 1 4 1 3 

12 II 

'2 / 



(51) 



the Kronecker product of 4 with any of the singlet representations is given by 

4 x 1- = 4 , 



(52) 
(53) 



and the non-trivial Kronecker product of 4 x 4 is 

{4x4} = l 1 ©l 2 ®4 s e 4 5 [4 x 4] = 1 3 © 1 4 © 4 A , 

where { } denotes symmetrisation and [ | - antisymmetrisation. The other group theoretical 
details of SG(20,3) are summarised in App. A. 

Note that SG(20,3) does not contain any subgroup with an irreducible representation 
3 because 20 is not divisible by 3. Therefore it cannot be broken down to 3 + 1, and con- 
sequently specific properties of the 4th generation compared to the three other generations 
cannot be explained as immediate consequence of the symmetry breaking. 



4.3 Flavour Structures and SG(20,3) 

Let us find possible flavour structures (structures of the fermion mass matrices) which can 
be obtained with SG(20, 3) symmetry. The required Clebsch-Gordan coefficients are given 
in App. A. 

1) In the limit of exact symmetry, the operators which lead to fermion masses have 
the form mFiF 2 . (We omit the usual non-flavoured Higgs fields, which should be added 
to satisfy gauge invariance.) Here F% and Fi are fermion multiplets transforming under a 
certain representation of SG(20,3). If Fi form quartets, Fi ~ 4, the mass operators has 
group structure 4x4 and leads to the symmetric non-singular mass matrix: 



/ 





m 





\ 











m 




m 













\ 


in 








/ 



(54) 



Apparently it cannot be used for RH neutrinos. If Fi are singlets of SG(20,3), Fi ~ 1]_ © 
12 © I3 © I4 the mass operators, (l x © 1_ 2 © I3 © I4) x (l x © 1_ 2 © I3 © I4), generate 
the mass matrix 





mi 


















m 2 





















777.4 












m 3 





/ 



(55) 
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If one of the mass parameters vanishes, m; = 0, this matrix has a vanishing eigenvalue. It 
can be used to describe three massive and one massless Majorana RH neutrinos. In the case 
Fx = F 2 = N, the matrix (55) is symmetric and the condition m\ = or m 2 = should be 
satisfied. 

2) Let us consider operators of the type yF\F 2X with flavon fields X which transform 
non-trivially under SG(20,3). They generate the mass terms, when the flavour symmetry 
is broken: (%) ^ 0. We introduce the quartet flavons, (ft ~ 4, and the singlets x% ~ li, with 
i = 1,2,3,4, and denote the VEVs of these fields as (xi) = (lj) = «i and ((f)) = (4) = 
(t>i, v 2 , v 3 , vA. If Fi,F 2 ~ 4, then the following invariant operators can be introduced: 



2y j {F 1 F 2 ) jXj J = 2,3,4, ^{F^j^, ^2/52 {^W, V^/a^ 



(56) 



where j/j are the Yukawa couplings, and in the second operator there are three different 
possibilities of pairing. (The operator with j — 1 gives the structure (55) without symmetry 
breaking.) The operators (56) produce the matrix 



V2IJS1 
MVS2 - Va) 
u 2 y 2 - u 3 y 3 - U 4 ]J4 
Mys2 + Va) 



HF 1 ,F 2 



v±{ys2 -Va) \ 
-u 2 y 2 - iu 3 y 3 + iu 4 y 4 

v?Xys2 + VA) 

viysi j 
(57) 

=1 lj are 4 different singlets of the symmetry group, then the symmetry 

=1 1: and only singlet flavon 



Vl(yS2 + VA) 
V3VS1 

V2{ys2 - Va) 
i (u 3 y 3 - u A y 4 ) - u 2 y 2 



u 2 yi + u 3 y 3 + w 4 ?/4 
v2(ys2 + yA) 

v^ysi 
v 3 (ys2 - Va) 



structure of the fermionic part of the operator is 
fields can be used. The invariant combinations 



yi{F 1 ) 2 (F 2 )iX2, 

2/5(^1)4(^2)1x3, 
2/9(^1)3(^2)1x4, 

generate the mass matrix 



2/2(^1)1(^2)2X2, 2/3(^1)3(^2)3X2, 
2/6(^1)3(^2)2X3, 2/7(^1)2(^2)3X3, 
2/10(^1)4(^2)2X4, 2/11(^1)1(^2)3X4, 



/ 

M22/1 
M42/9 
\ u 3 y 5 



u 2 y 2 


«32/6 
M42/10 



UiVll %2/8 \ 

M32/7 M42/12 

u 2 y 3 

M22/4 / 



2/4(^1)4(^2)4X2, 
y 8 (Fi)i(F 2 ) 4 X3, 
2/12(^1)2(^2)4X4 



(5? 



(59) 



The value of each matrix element is independent; and if only one flavon field X i is introduced 
(i.e only one U{ in the matrix above is non-zero) the symmetry only demands that four matrix 
elements are generated. Apparently there is no contribution from the flavon 0. Finally, if 



F\ ~ ©f =1 lj and F 2 ~ 4, the fermionic flavour structure 4x6 
flavons </>. The operators 

2/i{i 7 20}i(i 7 \)i, y2{F 2 <j)} 2 (F 1 ) 2 , y 3 [F 2 <l>MF 1 ) 3 , 



=1 1: requires the quartet of 

2/4[F20]3(i 7 l)4 (60) 
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produce the mass matrix 



2 



1 



/ «33/i 

V3V2 

\ v 3 y 4 



VaVi 
-ivm 



viyi 

-vm 
-vm 



V2V1 \ 
-V2V2 

-iv 2 y4 J 



(61) 



Several important conclusions can be drawn from the forms of these mass matrices. 
A singular Majorana mass matrix with only one vanishing mass eigenvalue cannot be ob- 
tained as an immediate result of the SG(20, 3) symmetry or its breaking without additional 
assumptions or symmetries. Such a matrix can be obtained by tuning of couplings which 
in turn requires introduction of additional symmetries. For example, suppose the RH neu- 
trinos transform as 4 and the direct mass terms are somehow forbidden so that the leading 
order contribution comes from the one flavon insertion ~ 4. Then according to Eq. (57) , 
one massless RH neutrino can be obtained with the VEV alignment (0) = v (1, 1, 1, 1), and 
equality ysi = ±1/52 of the Yukawa couplings of the two possible (symmetric {iViV}) invari- 
ants. A study of the simplest potential with one four-dimensional representation shows that 
the only allowed VEV configuration is indeed (0) = v(l, 1, 1, 1), which breaks SG(20,3) 
to Z4, as it is shown in Tab. 1, unless there are special relations between parameters in the 
flavon potential. 

4.4 Models and Phenomenology 

In the following, we discuss the leading order predictions for different symmetry assignments 
for the fields. Let us assign for leptons the following transformation properties: i ~ 4 and 
e R ~ I2 + I3 + I4 + ll > which allows to generate different charged lepton masses and 
explain the number of generation. We use flavons <fi ~ 4 and x% ~ lj- 

1) If the RH neutrinos transform as singlets N ~ I2 + 13 + I4 + ll ; the RH Majorana 
mass matrix has full rank. In order to obtain a singular RH neutrino mass matrix, we set 
M4 to zero (which according to our symmetry assignment corresponds to parameter mi in 
matrix (55)). This can be obtained in different ways: (i) by choice, (ii) by a "missing" 
representation, i. e. by choosing the assignment of representations of iV in such a way that 
there is only one of the complex conjugate representations I3 4 or (iii) by an additional 
symmetry, e.g. — > iiV 4 , which we are going to discuss in the following. This effectively 
leads to a 3 + 1 structure of the RH neutrinos and forbids several couplings in the Dirac 
mass matrices. Therefore, we demand the following transformation properties <fi — > — i<p, 
E4 — > 1E4 as well as introduce another flavon rj ~ 1^ transforming as 1] — > irj. All other 
fields are invariant under the additional symmetry. The leading order Lagrangian is given 
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by 



£ = MiJVf iVi + M 2 + N3N2) + YZlNtHij- + Y\lE 2 e m 



A 



+ YflNkH^ + Y l k iH 2 e kR ^ 4 h.o. 



(62) 



with k 
fi = 1, 



= 1,2,3. Here Y- (j = 1, 2, 3, 4) are the charged lepton Yukawa couplings and 
2, 3, 4) are the neutrino Yukawa couplings. This leads to the following 4x4 mass 



(63) 



The charged lepton mass matrix has the same structure as the Dirac neutrino mass matrix. 
In the basis, where the mass matrices of charged leptons and the RH neutrinos are 



matrices in the basis 


Ve,n,r,E and #1,2,3,4 


for ((f)) = 


V 


(1, 1, 


1, 1) and (77) = u 
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\ 






( 




V u y u 


Y 4 \ 
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M 2 
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-*Y 2 f *Y 3 f 


Y 4 


M 2 





2A 






-r 2x -r 3x 


Y 4 
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J 






K 






yJ 



fl 



diagonal: m 
Dirac neutrino mass matrix becomes 



2A 



diag(Y 1 / M/A, Y 2 l u/A, Y 3 l u/A, Yj), M fl = diag(Mi, M 2 , M 2 , 0), the 



m 



fi 



v v 



EW 



A 



I 1 A 




V 



V2A 
v^A 







v^A 
v^A 





\ 







Y? 



(64) 



/ 



where 



'EW 



(Hi) and v 



EW 



It is not diagonal like the charged lepton mass 
matrix due to the additional rotation from diagonalising the RH neutrino mass matrix. Its 
structure corresponds to fi = /r = in our general consideration of Sec. 2. Hence, the 
fourth generation decouples from the first three generations and the only contribution to 
the light neutrino mass matrix originates from the ordinary see-saw mechanism 



m 




(65) 



The coefficients A and B are given by A = —Y^ 



Mi A 2 



and B 



-Y 2 Y 3 



and lead to a vanishing atmospheric mass squared difference. 



M3A 2 ' 



respectively, 



This mass squared difference can be obtained by the introduction of flavon \2 ~ l2> 
whose different couplings N 2 N 2 X2 and N 3 T N 3 X2 split masses of the RH neutrinos N 2 and 
N 3 which otherwise equal M 2 (see M? 1 above). This split in turn generates non-zero ele- 
ments rrf^ and m s * T in the see-saw matrix (65). If flavon singlets Xk ~ !k with k = 2,3,4 
and (xk) = Uk are introduced, which transform under additional symmetry as Xk — > — iXfc, 
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mixing between the fourth generation and the three light generations is generated in the 
RH Majorana mass matrix by interactions iVj( ^2X2^1 + ^4X4^2 + ^3X3^3) with Yukawa 
couplings hk- Also these additional flavons contribute to the 3x3 block of the first three 
generations in the Dirac mass matrices because 0xj~ is invariant under the additional sym- 
metry. This leads to appearance of Jl,r as well as M4 due to the mixing of N4 with N^. 
Hence, there are the rank 1 two loop contribution and the tree level contribution to neutrino 
masses (see in Sec. 2). As the tree level contribution is generated at a higher order in flavon 
insertions compared to the see-saw contribution, it can be neglected at leading order. 

Let us comment on other possible VEV alignments and structure of the neutrino Dirac 
mass matrix. Any VEV alignment of the quartet, which differs from equality of components, 
induces mixing between the fourth neutrino and the three light ones, which is proportional 
to the deviation from the VEV alignment (0) = v (1, 1, 1, 1) besides generating the elements 
m^ s M and m s ^ T . However, the constraints on the mixing between the fourth and the three light 
SM generations does not allow large enough values for m^ s „ and m s ^ T without introducing 
additional flavons Xk ~ l^- 

Summarising, the simplest construction with only one flavon ~ 4 does give correct 
values of neutrino masses and mixing. 

2) On the contrary, suppose the RH neutrinos transform as N ~ 4 and the direct mass 
term NN is forbidden by an additional auxiliary symmetry, e.g. N — > tuN with u = e 27rl//3 
in order to achieve a singular RH neutrino mass matrix, the Lagrangian is given by 

- £ = ^i{V T V W + MiV T V} off _ diag + Y^lN^H^ + Yi£H 2 e iR ^- + h.c. , (66) 

where ~ 4 and — > ucf> as well as tiR —> ujtiR. Y£ {k = SI, S2, A) correspond to three 
possible combinations of (£N) which transform as 4 5 , 4 5 , 4 A (see Eq. (53)). Taking the 
VEV alignment (0) = v (1, 1, 1, 1), the RH neutrino mass matrix becomes 



M 



f hi h 2 h 2 \ 
... hi h 2 

h h 2 



(67) 



with mass eigenvalues \hi\v, \hi\v, \hi — 2h 2 \v and \hi + 2h 2 \v. Hence, there is exactly 
one massless RH neutrino if hi = ±2h 2 . Under the assumption hi = —2h 2 , which can be 
obtained by fine-tuning the couplings hip only, the diagonalised RH neutrino mass matrix 
becomes M^ 1 = \hi\v diag(2, 1, 1,0). The eigenstate corresponding to the zero mass eigen- 
value is (1, 1, 1, 1). In flavour basis, where the charged leptons and the RH neutrino mass 
matrix are diagonal, the charged lepton mass matrix is = 2 ^ w diag(V 1 e , Y 2 e , Y 3 e , V 4 e ) 
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and the Dirac neutrino mass matrix is given by 



/ - Y, 



m 



fi 



v v 



EW 



2A 



V 

S2 



\ 















2YX+iYg 2 




V2 




-2YX+iY£ 2 


-2iYJ-Y» 2 


V2 

















Z1 SI 
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7 



(6* 



This leads to a see-saw contribution to the neutrino mass matrix of the form (65) with 



coefficients A 



(Y S2 -2Y S1 ) 2 



and B 



Hence, it has the same structure with 



2hi ' hi 

different coefficients and, essentially, the same conclusions can be drawn as in case f). 
Similarly to the previous assignment of representations, the addition of flavons Xk ~ !k 
will lead to a non-vanishing atmospheric mass squared difference and a mixing of the fourth 
generation with the first three SM generations. 

Concluding, the simplest group SG(20,3) does not immediately lead to the required 
flavour structure. For instance tuning of Yukawa couplings is required to obtain zero mass 
for one of the RH neutrinos if they transform as 4. In this case, the direct mass term of a four 
dimensional representation always has full rank, and the RH neutrino mass matrix has to be 
generated using non-singlet flavons. Furthermore, an additional leading order contribution 



to the neutrino mass matrix is required which generates the ( 
have demonstrated how those contributions can be achieved. 



and 



m,, 



entries. We 



A survey of all small groups up to order 56 with the RH neutrinos transforming as 4 
shows that the RH neutrino masses are either all of the same order of magnitude or there 
are two heavy and two light RH neutrinos. Hence, one might argue that 4 is generally not 
the best representation for the RH neutrinos and a more viable choice is 3© 1', where 3 is a 
real representation of a given group and V a complex representation such that the matrix of 
direct mass terms is singular with one vanishing mass. This splitting might also be obtained 
from breaking the symmetry group to a smaller subgroup. The smallest group, which allows 
the decomposition of 4 into 3 © 1, is A5. However A 5 has irreducible representations 3 as 
well as 5, so that use of representation 4 only should be justified. 



5 Summary and Conclusions 

I. We have explored the generation of light neutrino masses in the presence of a fourth 
family of fermions with four RH neutrinos (1 per family). In this context, generically there 
are three contributions to the light neutrino masses from three different mechanisms: 

(i) the usual high mass scale see-saw contribution; 

(ii) the tree level contribution induced by mixing of the light generations with the fourth 
generation. This contribution requires mixing of both left and right neutrino com- 
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ponents (fi 7^ fn 7^ 0) in the basis where the Majorana mass matrix of the RH 
components is diagonal. 

(iii) The two loop contribution with two PF-bosons exchange, related to the non-zero Ma- 
jorana mass of the fourth neutrino, M 4 . 

2. We show that even if M4 = initially at tree level, it will be generated at the two 
loop level due to usual Yukawa interactions. This radiatively generated mass is proportional 
to the large Majorana masses Mj and therefore, is rather large: 0.1 — 1 GeV. Unless there 
is strong cancellation, (e.g., with tree level contribution), this mass, in turn, produces the 
dominant contribution to the light neutrino masses in large part of parameter space. In the 
case fn = 0, the new contributions related to the fourth generation vanish. The relative 
contributions from different mechanisms depend strongly on M, m, £ = {U^aiiU^a , and 
to a smaller degree on and 777.4. The tree level contribution of the fourth generation 
dominates over the loop contribution for small RH neutrino masses M and vice versa, as it is 
illustrated in Figs. 3 and 4. The smaller Dirac mass, m, the larger the tree level contribution 
compared to the usual see-saw contribution; the loop contribution is independent of m. The 
usual see-saw contribution does not depend on £, while the tree level contribution and the 
loop contribution are proportional to £ and £ 2 respectively. 

3. In general, the contributions from three different mechanisms have different flavour 
structures. The loop contribution is singular and therefore it cannot explain the observed 
mass hierarchy. Therefore, comparable contributions should follow from other mechanisms. 
The combination of the loop and see-saw contributions is realized at Mj ~ (10 7 — 10 10 ) GeV, 
a large Dirac neutrino mass m, and £ = 1CT 9 — 10~ 7 . Combination of the loop and tree level 
contributions works for Mi < 10 5 GeV (and therefore small m) and £ = 10~ 6 . An interplay 
of the "see-saw and tree level" contributions is realized at small Dirac neutrino mass m and 
£ < 10~ 7 . The loop contribution gives very strong bound on mixing parameters especially 
for large Mj. At Mj > 10 4 GeV the bounds are much stronger than those from the direct 
searches. The loop contribution can be suppressed if certain cancellation occurs between 
contributions from different Mj or between the loop and tree level contribution, although 
this looks rather unnatural. 

The upper bound on the light neutrino masses (which follows from cosmology) gives the 
most stringent bound on the parameters of the model. 

4. In the see-saw limit of the fourth generation, where the Majorana mass M4 is much 
larger than the Dirac mass 1714, M4 3> , the tree- level contribution of the fourth genera- 
tion to the three light active neutrinos is negligible. In the pseudo-Dirac limit, m 4 3> M 4 , the 
tree-level contribution of the 4th generation to the light neutrino mass matrix is significant 
for small RH neutrino masses Mj and determines one of the mass scales. 

5. We explored flavour symmetries, which could explain the leptonic flavour structure 
and studied the smallest group with a four-dimensional representation, SG(20, 3) = Z 5 xi ¥ ,Z 4 . 
We have found the simplest flavour structures (mass matrices), which can be obtained as a 



26 



result of this flavour symmetry. The required singularity of the RH neutrino mass matrix 
can be achieved imposing conditions on the Yukawa couplings and VEVs. There is no viable 
model based on one flavon ~ 4, but there are phenomenologically viable models can be 
constructed with flavons <fi ~ 4 and Xk ~ Ik- We indicated the next smallest groups, which 
might be interesting to study. 
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A Group Theory of SG(20, 3) ^ Z 5 Z 4 

The generators of the four-dimensional representation and the character table are given in 
Tab. 1. The Clebsch-Gordan coefficients for the Kronecker product 4x4 can be calculated 
following the algorithm described in [68]. The Kronecker product of a = {a±, a 2 , a 3 , en) ~ 4 
and b — {pi, b 2 , b 3 , 64) ~ 4 results in 

II ~ \ ( a ih + °3&i + + a^h) , ±2 ~ \ ( fl i & 3 + a>zh - a 2 b /L - a 4 b 2 ) , (69) 

I3 ~ I ( a i & 3 - a 3 bi + i a 2 & 4 - i a 4 b 2 ) , I4 ~ - (^63 - a 3 bi - i a 2 b 4 + i a 4 6 2 ) , 

4 S ~ —= (a 3 6 4 + a 4 b 3 , a 4 &i + aib 4 , a x b 2 + a 2 b\, a 2 b 3 + a 3 b 2 ) , (70) 
V2 

4 S ~ (a 2 b 2 , a 3 b 3 , a^b A , a x bi) , 

4 A ~ — (a 3 6 4 — a 4 6 3 , a 4 6i — a,\b±, a\b 2 — a 2 b\, a 2 b 3 — a 3 b 2 ) . 
V2 

The Clebsch-Gordan coefficients for 1 • x 4 with a ~ 1 • and {b\, b 2 , b 3 , 6 4 ) ~ 4 are 

1]_x4~(q!6i, a b 2 , ab 3 , a 64) , 1 2 x 4 — [ab\, —ab 2 , ab 3 , —a 64) , (71) 

I_3x4~(a6i, — iab 2 , —ab 3 , iabA , L4 x 4 ~ (a bi, iab 2 , —ab 3 , — iafe 4 ) , 
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(a) Character table (b) Breaking patterns 





classes 




Ci 


c 2 




c 4 


c 5 


G 


1 


s 


s 2 




s 3 


he, 


1 


4 


2 


5 


4 


II 


1 


1 


1 


1 


1 


12 


1 


-1 


1 


1 


-1 


13 


1 


— i 


-1 


1 


i 


U 


1 


i 


-1 


1 


— i 


4 


4 








-1 






subgroup 


VEV configuration 


D 5 


(l2> 


z 5 


(Is) 








z 5 


(l4> 








z 4 


(4) ~ (1, 


1, 1, 1) 






z 4 


(4) ~ (t? 4 , 


77, r? 2 , 1) 






z 4 


(4) ~ (t? 3 , 


r/ 2 , 7] 4 , 1) 






z 4 


(4) ~ {v 2 , 


?7 3 , 77, l) 






z 4 


(4) ~ (v, 


i] 4 , i] 3 , l) 






z 2 


(4) ~ (1, 


0, 1, 0), (0, 


1, 


0, 1) 


z 2 


(4) ~ {v 2 , 


0, 1, 0), (0, 


V, 


0, 1) 


z 2 


(4) ~ {v\ 


0, 1, 0), (0, 




0, 1) 


z 2 


(4) ~ (77, 


0, 1, 0), (0, 




0, 1) 


z 2 


(4) ~ (t? 3 , 


0, 1, 0), (0, 


v 4 


0, 1) 



(c) Generators of 4 



/ 1 \ / 77 \ 

OOfO 0r7 4 00 

S ~ fOOO ?? 2 

\ 1 / \ Or? 3 / 

Table f: Group theoretical details of SG(20,3): Character table in Tab. (a), breaking 
patterns in Tab. (b), and the generators of 4 in Tab. (c). G denotes the generating element, 
is the order of the elements and 77 = exp(27ri/5) is the fifth root of unity. 
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where S and A indicate that the representation is in the symmetric or antisymmetric part, 
respectively. The different possible breaking patterns of SG(20, 3) to its subgroups are shown 
in Tab. 1, where we have used the algorithm described in Appendix C of [69]. 
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